Imaging device and image processing method

ABSTRACT

According to the present invention, even if variation (estimation variation) is caused in a subject distance estimated at the time of imaging when restoration processing of a taken image is performed using a restoration filter corresponding to the subject distance, since an optimal restoration filter that conforms to a subject distance taking into account the maximum infinity-side estimation variation and does not cause overcorrection is selected, it is possible to prevent resolution degradation due to overcorrection and achieve the improvement of resolution by the restoration filter.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of PCT International Application No.PCT/JP2013/061841 filed on Apr. 23, 2013, which claims priority under 35U.S.C. §119(a) to Japanese Patent Application No. 2012-213830 filed onSep. 27, 2012. Each of the above application(s) is hereby expresslyincorporated by reference, in its entirety, into the presentapplication.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an imaging device and an imageprocessing method, and specifically relates to an imaging device and animage processing method that perform point restoration processing of animage taken on the basis of PSF (Point Spread Function).

2. Description of the Related Art

Point restoration processing of a taken image is processing of findingthe characteristic of degradation (PSF) due to the aberration or thelike of a photographing lens beforehand and restoring the taken image(degraded image) to an original image of high resolution by performingfilter processing by the use of a restoration filter created on thebasis of the PSF.

PTL 1 (Japanese Patent Laid-Open No. 2011-211663) discloses an imageprocessing apparatus that acquires an imaging state including a subjectdistance, acquires a restoration filter corresponding to the imagingstate of each restoration region of a degraded image and restores thedegraded image of each restoration region by the use of the degradedimage and the restoration filter. Here, the PSF varies according to thelens type, zoom magnification, the angle of view (image height) and adiaphragm besides the subject distance, and therefore a lot ofrestoration filters are prepared.

Moreover, a subject distance calculation unit described in PTL 1calculates a distance (subject distance) to the subject in a focusposition and the subject distance in each focus detection region, from aphase difference in multiple focus detection regions and focus lenspositions at the time of focus detection and focusing.

Meanwhile, PTL 2 (Japanese Patent Laid-Open No. 2005-173254) contains adescription that an error is caused between a detected subject distanceand an actual subject distance if a photographing lens is transformed bytemperature when a subject distance is calculated on the basis of a lensposition, and it describes a camera system that finds this error bydetecting temperature.

SUMMARY OF THE INVENTION

The image processing apparatus described in PTL 1 acquires an imagingstate including a subject distance and performs restoration filterprocessing of a degraded image by the use of a restoration filtercorresponding to the imaging state.

However, if an inappropriate restoration filter is used as a restorationfilter used for restoration filter processing, there is a problem ofcausing overcorrection and deteriorating the resolution contrarily.

As a condition that the overcorrection is caused, a case is possiblewhere the subject distance is measured in an incorrect manner and, as aresult, a restoration filter corresponding to the incorrect subjectdistance is used.

Since the aberration of a lens varies according to a focus position(subject distance), it is necessary to apply a restoration filtercorresponding to the subject distance. Regarding the subject distance, amethod of finding it from the focusing stop position of a focus lens atthe time of auto focus (AF) is used, but there is a problem that, whenthe characteristic of a lens barrel varies by temperature, the focusingstop position of the focus lens varies even if the subject distance isconstant, and incorrect measurement is performed.

Meanwhile, it is considered that an error between a subject distance andan actual subject distance is found by detecting the temperature of acamera like the camera system described in PTL 2, but there is a limitto accurately find the subject distance from the focusing stop positionof the focus lens. For example, in AF of a contrast detection scheme ora phase difference detection scheme, it is considered that the accuracyof focusing determination decreases in a case where the contrast of thesubject is low, and the reliability of the focusing stop position of thefocus lens is low.

Moreover, in a case where an interchangeable lens is applied as animaging lens, since the temperature characteristic varies depending onthe kind of the interchangeable lens and there are individualdifferences, it is difficult to accurately find the subject distancefrom the focusing stop position of the focus lens.

The present invention is made in view of such circumstances, and it isan object to provide an imaging device and image processing method thatcan: prevent resolution degradation due to overcorrection even ifvariation (estimation variation) is caused in a subject distanceestimated at the time of imaging when restoration processing of a takenimage is performed using a restoration filter corresponding to thesubject distance; and achieve the improvement of resolution by therestoration filter.

An imaging device according to one mode of the present invention toachieve the above-mentioned object includes: an imaging unit configuredto take a subject image formed by an imaging lens and acquire an imageshowing the subject image; a first subject distance estimation unitconfigured to calculate an estimated subject distance by a focusdetection unit at imaging by the imaging unit; a restoration filterstorage unit configured to store multiple restoration filters createdbased on at least a point spread function corresponding to a subjectdistance; an estimation variation acquisition unit configured to acquirea maximum infinity-side estimation variation in a range of estimationvariation with respect to the subject distance estimated by the firstsubject distance estimation unit; a second subject distance estimationunit configured to calculate a subject distance adding the maximuminfinity-side estimation variation acquired by the estimation variationacquisition unit to the subject distance estimated by the first subjectdistance estimation unit; a restoration filter selection unit configuredto select a restoration filter corresponding to an infinity side closestto the subject distance calculated by the second subject distanceestimation unit among the multiple restoration filters stored in therestoration filter storage unit; and a restoration processing unitconfigured to perform restoration processing of the image acquired bythe imaging unit using the restoration filter selected by therestoration filter selection unit.

According to one mode of the present invention, the subject distance isestimated by the first subject distance estimation unit, and the maximuminfinity-side estimation variation in the range of variation (estimationvariation) with respect to this estimated subject distance is acquiredby the estimation variation acquisition unit. A subject distance addingthe subject distance estimated by the first subject distance estimationunit and the maximum infinity-side estimation variation acquired by theestimation variation acquisition unit is calculated. This calculatedsubject distance is a subject distance when the estimated subjectdistance varies on the infinity side most. Further, when a restorationfilter used for restoration processing is selected from multiplerestoration filters which are stored in the restoration filter storageunit and which correspond to the subject distance, a restoration filtercorresponding to the infinity side closest to the calculated subjectdistance is selected.

A point spread function (PSF) varies according to the subject distance,and a PSF on the close end side has a wider shape than a PSF on theinfinity side. As a result, the restoration filer on the close end sideis prepared in advance as the one to perform stronger restorationprocessing, as compared with the restoration filter on the infinityside. By selecting a restoration filter corresponding to the infinityside closest to the subject distance when the estimated subject distancevaries on the infinity side most, it is possible to select a restorationfilter corresponding to the subject distance when it varies on theinfinity side most, or a restoration filter to perform weakerrestoration processing than it. By this means, it is possible to preventrestoration processing of overcorrection by the selected restorationfilter (prevent resolution degradation) and achieve the improvement ofresolution by the restoration filter.

In an imaging device according to another mode of the present invention,it is preferable that the first subject distance estimation unitestimates the subject distance based on a lens position of a focus lensof the imaging lens.

In an imaging device according to another mode of the present invention,it is preferable that: the imaging lens is an interchangeable lens; andthe estimation variation acquisition unit has an estimation variationstorage unit configured to store the maximum infinity-side estimationvariation of the interchangeable lens for each interchangeable lens,acquires information on the interchangeable lens to be attached, andacquires a corresponding estimation variation from the estimationvariation storage unit based on the acquired information on theinterchangeable lens. An estimation variation of the subject distanceestimated by the kind of the interchangeable lens (high-level lens andmedium-level lens, and so on) varies, but, in this case, information onthe attached interchangeable lens is acquired and a correspondingestimation variation is acquired from the estimation variation storageunit on the basis of the information on the interchangeable lens.

In an imaging device according to another mode of the present invention,it is preferable that: the imaging lens is a lens having an estimationvariation storage unit configured to store the maximum infinity-sideestimation variation in the range of estimation variation with respectto the subject distance estimated by the first subject distanceestimation unit; and the estimation variation acquisition unit acquiresthe estimation variation from the estimation variation storage unit ofthe lens. By this means, it is possible to acquire the estimationvariation from the estimation variation acquisition unit on the lensside without providing the estimation variation storage unit on the sideof the imaging device body.

In an imaging device according to another mode of the present invention,the maximum infinity-side estimation variation in the range ofestimation variation with respect to the subject distance estimated bythe first subject distance estimation unit is at least one of estimationvariation by an individual difference of an interchangeable lens andestimation variation by a temperature characteristic of theinterchangeable lens.

In an imaging device according to another mode of the present invention,it is preferable that the restoration filter storage unit storesmultiple corresponding restoration filters in a range between aninfinity and a subject distance adding only the maximum infinity-sideestimation variation in the range of estimation variation with respectto a nearest-side subject distance to the nearest-side subject distanceestimated by the first subject distance estimation unit. A restorationfilter within a range on the nearer side than a subject distance(hereafter referred to as “estimation variation close range”) adding themaximum infinity-side estimation variation in the range of estimationvariation with respect to the nearest-side subject distance to thenearest-side subject distance estimated by the first subject distanceestimation unit, may perform restoration processing of overcorrection.Therefore, by storing only multiple corresponding restoration filters inthe range between the estimation variation close range and the infinitysuch that a restoration filter within a range on the nearer side thanthe estimation variation close range is not provided from the beginning,the number of restoration filters stored in the restoration filterstorage unit is reduced.

In an imaging device according to another mode of the present invention,it is preferable that the restoration filter storage unit stores themultiple restoration filters corresponding to a subject distanceregardless of the imaging lens. By this means, it is possible to sharemultiple restoration filters regardless of the imaging lens.

In an imaging device according to another mode of the present invention,it is preferable that the imaging lens is an interchangeable lens inwhich the restoration filter storage unit is incorporated. By thismeans, it is possible to acquire a restoration filter corresponding tothe lens from the restoration filter storage unit on the lens sidewithout providing the restoration filter storage unit on the side of theimaging device body.

In an imaging device according to another mode of the present invention,it is preferable that a recording unit configured to record an imageacquired by performing restoration processing by the restorationprocessing unit is included.

In an imaging device according to another mode of the present invention,it is preferable that: the restoration filter storage unit has multipletables in which multiple restoration filters are stored corresponding tothe estimation variation; and the restoration filter selection unitselects a corresponding table from the multiple tables according tomagnitude of the estimation variation acquired by the estimationvariation acquisition unit, and selects the restoration filter from theselected table.

An imaging method according to another mode of the present inventionincludes: an image acquisition step of acquiring an image showing asubject image from an imaging unit; a first subject distance estimationstep of calculating an estimated subject distance by a focus detectionunit at imaging by the imaging unit; a step of preparing a restorationfilter storage unit configured to store multiple restoration filterscreated based on at least a point spread function corresponding to asubject distance; an estimation variation acquisition step of acquiringa maximum infinity-side estimation variation in a range of estimationvariation with respect to the subject distance estimated in the firstsubject distance estimation step; a second subject distance estimationstep of calculating a subject distance adding the maximum infinity-sideestimation variation acquired in the estimation variation acquisitionstep to the subject distance estimated in the first subject distanceestimation step; a restoration filter selection step of selecting arestoration filter corresponding to an infinity side closest to thesubject distance calculated in the second subject distance estimationstep among the multiple restoration filters stored in the restorationfilter storage unit; and a restoration processing step of performingrestoration processing of the image acquired by the imaging unit usingthe restoration filter selected in the restoration filter selectionstep.

According to the present invention, even if variation (estimationvariation) is caused in a subject distance estimated at the time ofimaging when restoration processing of a taken image is performed usinga restoration filter corresponding to the subject distance, since anoptimal restoration filter that conforms to a subject distance takinginto account the maximum infinity-side estimation variation and does notcause overcorrection is selected, it is possible to prevent resolutiondegradation due to overcorrection and achieve the improvement ofresolution by the restoration filter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an outline drawing of an imaging device according to one modeof the present invention.

FIG. 2 is a back view of the imaging device illustrated in FIG. 1.

FIG. 3 is a main part block diagram of the imaging device illustrated inFIG. 1.

FIG. 4 is a diagram illustrating the flow of contrast auto focusprocessing.

FIG. 5 is a diagram illustrating contrast auto focus.

FIG. 6 is a main part block diagram of a focus lens control unit.

FIG. 7 is a main part block diagram of a main CPU and digital signalprocessing unit that perform point restoration processing.

FIG. 8 is a diagram illustrating the way of storing and selecting arestoration filter in the first embodiment of the present invention.

FIG. 9 is a diagram illustrating the way of storing and selecting arestoration filter in the first embodiment of the present invention.

FIG. 10 is a diagram illustrating a way of storing and selecting arestoration filter in the first embodiment of the present invention.

FIG. 11 is a diagram illustrating the way of storing and selecting arestoration filter in the second embodiment of the present invention.

FIG. 12 is a diagram illustrating the way of storing and selecting arestoration filter in the second embodiment of the present invention.

FIG. 13 is a diagram illustrating the way of storing and selecting arestoration filter in the second embodiment of the present invention.

FIG. 14 is an outline drawing of a smartphone that is another embodimentof the imaging device.

FIG. 15 is a block diagram illustrating a main part configuration of asmartphone.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1 is a perspective view illustrating the appearance of an imagingdevice 100 according to one mode of the present invention. The imagingdevice 100 includes an imaging device body 200 and a lens apparatus 300attached to the imaging device body 200 in an exchangeable manner. Theimaging device body 200 and the lens apparatus 300 are attached in anexchangeable manner by combining a mount 246 (transmission means andreception means) included in the imaging device body 200 and a mount 346(reception means and transmission means) (see FIG. 3) on the side of thelens apparatus 300 corresponding to the mount 246. Moreover, besides themount 246, a flash 240 is installed in the front surface of the imagingdevice body 200, and a release button 220-1 and a dial 220-2 for thesetting of an imaging mode are installed in the upper surface.

A terminal 247 (transmission means and reception means) is installed inthe mount 246, a terminal 347 (transmission means and reception means)(see FIG. 3) is installed in the mount 346, and, when the lens apparatus300 is attached to the imaging device body 200, the correspondingterminal 247 and terminal 347 are contacted and communication becomespossible. Here, the terminal 247 and the terminal 347 in FIGS. 1 and 3are conceptually illustrated, and the positions and number of terminalsin the present invention are not limited to the ones in these figures.

Meanwhile, as illustrated in FIG. 2, a monitor 212, a cross button 260,a MENU/OK button 265, a reproduction button 270 and a BACK button 275,and so on, are arranged in the back surface of the imaging device body200. Here, the imaging device 100 according to one mode of the presentinvention may be either a lens-exchange-type imaging device or alens-fixed-type imaging device.

FIG. 3 is a block diagram illustrating the configuration of the imagingdevice 100. The operation of the imaging device 100 is integrallycontrolled by a main CPU 214 of the imaging device body 200 and a lensCPU 340 of the lens apparatus 300. A program (including a program todrive zoom lens ZL focus lens FL and diaphragm I) and data required forthe operation of the main CPU 214 are stored in a flash ROM 226 and aROM 228 in the imaging device body 200, and a program (including aprogram to drive zoom lens ZL, focus lens FL and diaphragm I) and datarequired for the operation of the lens CPU 340 are stored in a ROM 344in the lens CPU 340.

Besides the release button 220-1 and the dial 220-2, an operation unit220 including a reproduction button, a MENU/OK key, a cross key and aBACK key, and so on, are installed in the imaging device body 200, and,by operating the buttons and keys included in the operation unit 220,the user can give an instruction such as the selection of aphotographing/reproduction mode, the start of photographing, theselection/reproduction/deletion of an image and a zoom instruction.

A signal from the operation unit 220 is input in the main CPU 214, andthe main CPU 214 controls each circuit of the imaging device body 200 onthe basis of the input signal and transmits/receives a signal to/fromthe lens apparatus 300 through the mount 246 (transmission means andreception means) and a mount communication unit 250 (transmission meansand reception means).

For example, the above-mentioned terminals include an earth terminal, asynchronization signal terminal, a serial communication terminal, acontrol status communication terminal and a current supply terminal froma battery 242 of the imaging device body 200 to each part of the lensapparatus 300.

At the time of imaging, subject light is formed on the light receivingsurface of an imaging element 202 of the imaging device body 200 throughzoom lens ZL, focus lens FL (imaging lens) and diaphragm I of the lensapparatus 300. The imaging element 202 is assumed to be a CMOS type inthe present embodiment, but it is not limited to the CMOS type and itmay be a CCD type. Focus lens FL, zoom lens ZL and diaphragm I aredriven by a zoom lens control unit 310, focus lens control unit 320(lens drive means) and diaphragm control unit 330 that are controlled bythe lens CPU 340, and focus control, zoom control and diaphragm controlare performed.

The zoom lens control unit 310 moves zoom lens ZL in the optical axisdirection and changes the imaging magnification according to aninstruction from the lens CPU 340. Moreover, the focus lens control unit320 performs operation to move focus lens FL backward and forward in theoptical axis direction and focus it on the subject according to aninstruction from the lens CPU 340. The diaphragm control unit 330changes the diaphragm value of diaphragm I according to an instructionfrom the lens CPU 340.

When the release button 220-1 is pressed in the first stage (halfpress), the main CPU 214 starts AF and AE operation, and, according tothis, image data output from an A/D converter 204 is imported in anAE/AWB detection unit 224. The main CPU 214 calculates the brightness(imaging EV value) of the subject from an integrated value of a G signalinput in the AE/AWB detection unit 224, and, based on the result,controls the diaphragm value of diaphragm I, the charge storage time(corresponding to the shutter speed) in the imaging element 202 and theluminescence time of the flash 240, and so on.

An AF detection unit 222 is a part that performs contrast AF processingor phase difference AF processing. In a case where the contrast AFprocessing is performed, focus lens FL in a lens barrel is controlledsuch that the AF evaluation value that shows the focusing state and iscalculated by integrating the high-frequency component of image data ina focus region becomes maximum. Moreover, in a case where the phasedifference AF processing is performed, focus lens FL in the lensapparatus 300 is controlled such that the amount of defocus found fromphase difference data calculated using pixels having multiple phasedifferences in the focus region of the image data becomes 0. Here, theAF detection unit 222 moves as a focus detection unit that detectsfocus.

When the AE operation and the AF operation end and the release button220-1 is pressed in the second stage (full press), the flash 240 emitslight by control through a flash control unit 238. Moreover, a signalcharge accumulated in the imaging element 202 is read out as a voltagesignal corresponding to the signal charge on the basis of a readoutsignal added from an imaging element control unit 201, and added to ananalog signal processing unit 203. The analog signal processing unit 203samples and holds R, G and B signals of each pixel by correlation doublesampling processing for the voltage signal output from the imagingelement 202, amplifies them and then adds them to the A/D converter 204.The A/D converter 204 converts analog R, G and B signals to besequentially input into digital R, G and B signals and outputs them toan image input controller 205. Here, in a case where the imaging element202 is a CMOS imaging element, the A/D converter 204 is often built intothe imaging element 202, and there is a case where the above-mentionedcorrelation double sampling is not necessary.

Image data output from the image input controller 205 is input in adigital signal processing unit 206, subjected to signal processing suchas offset processing, gain control processing including white balancecorrection and sensitivity correction, gamma correction processing,restoration processing and YC processing, encoded in a display controlunit 210 and output to the monitor 212 after writing/reading to/from aVRAM 230, and thereby a subject image is displayed on the monitor 212.

Moreover, a recording unit that is means for recording theabove-mentioned acquired subject image may be included. For example, animage acquired by restoration processing may be recorded in a memorycard 236.

Moreover, image data output from the A/D converter 204 in response tothe full press of the release button 220-1 is input from the image inputcontroller 205 into an SDRAM (memory) 232 and temporarily stored. Afterit is temporarily stored in the SDRAM 232, an image file is generatedthrough signal processing such as gain control processing, gammacorrection processing and YC processing in the digital signal processingunit 206 and compression processing into the JPEG (Joint Photographiccoding Experts Group) format or the like in a compression/extensionprocessing unit 208, and the image file is read out by a media controlunit 234 and recorded in the memory card 236. The image recorded in thememory card 236 can be reproduced and displayed on the monitor 212 byoperating the reproduction button of the operation unit 220.

Next, communication between the imaging device body 200 and the lensapparatus 300 is described. The imaging device body 200 and the lensapparatus 300 perform communication through the mount 246 (transmissionmeans and reception means) and the mount communication unit 250(transmission means and reception means) of the imaging device body 200,the mount 346 (transmission means and reception means) and a mountcommunication unit 350 (transmission means and reception means) of thelens apparatus and terminals installed in the mount 246 and the mount346, and a lens movement instruction is transmitted and received.

The lens movement instruction includes a control target (zoom lensZL/focus lens FL/diaphragm I), a drive mode, a numerical value (thetarget position of zoom lens ZL/focus lens FL, and the diaphragm valueof diaphragm I, and so on), and brake ON/OFF information (informationshowing whether to short the coil of a motor 326 in the target positionand apply a brake). Besides the above, communication of various controlstatuses (lens drive start/completion notice, and so on) is performedbetween the imaging device body 200 and the lens apparatus 300 through acontrol status communication terminal.

<Contrast AF>

Next, contrast AF is described with reference to FIGS. 4 and 5.

FIG. 4 illustrates a flow chart of contrast AF processing. When thecontrast AF processing starts, first, the main CPU 214 outputs a lensmovement instruction that moves focus lens FL to a home position (HP),to the lens apparatus 300 in order to detect the lens position of focuslens FL, and moves focus lens FL to the HP (step S10).

When focus lens FL moves to the HP, the count value of an up-downcounter 320-8 (FIG. 6) is reset to make count value i equal to 0 (stepS12). Here, an HP sensor 322 (FIG. 6) such as a photo interrupter is setin the HP, and, when focus lens FL being moved based on the lensmovement instruction reaches the HP, the HP sensor 322 outputs a lensdetection signal. This lens detection signal is used as a reset signalthat resets the count value of the up-down counter 320-8 that functionsas lens position detection means to 0. Moreover, focus lens FL may bemoved to the HP when the lens apparatus 300 is attached to the imagingdevice body 200 or when a power supply is turned on.

Subsequently, the main CPU 214 outputs a lens movement instruction thatmoves focus lens FL from the infinity side (INF) to the close end (AFsearch) as illustrated in FIG. 5, to the lens apparatus 300 (step S14).

During the AF search, the main CPU 214 acquires the count value from theup-down counter 320-8 of the lens apparatus 300 as a lens positionsignal (step S16), acquires image data in a focus region for each properlens position and calculates the AF evaluation value in the AF detectionunit 222 (FIG. 3) (step S18). Here, the AF detection unit 222 calculatesthe AF evaluation value indicating the focusing state by integrating thehigh-frequency component of the image data in the focus region, andoutputs the calculated AF evaluation value to the main CPU 214.

The main CPU 214 decides whether the AF search ends (whether count valuei (focus lens FL position) is the close end (0.20)) (step S20), and, ina case where count value i is not the close end, it shifts to step S14to continue the AF search.

Meanwhile, when determining that the AF search ends, the main CPU 214calculates the lens position (focusing position) in which the AFevaluation value becomes maximum, on the basis of each AF evaluationvalue and the lens position which are acquired in steps S16 and S18(step S22 and (A) part of FIG. 5).

The main CPU 214 outputs a lens position instruction that moves focuslens FL to the calculated focusing position, to the lens apparatus 300,and moves focus lens FL to the focusing position (step S24 and (B) partof FIG. 5).

Moreover, the main CPU 214 calculates a subject distance from the lensposition calculated in step S22 (step S26).

<Focus Lens Control Unit>

FIG. 6 illustrates a functional block diagram of the focus lens controlunit 320. A lens movement instruction is sent from the main CPU 214 tothe focus lens control unit 320 through the lens CPU 340.

The focus lens control unit 320 mainly includes a controller 320-2, adriver 320-4, a motor 320-6 and the up-down counter 320-8. The lensmovement instruction from the lens CPU 340 is received by the controller320-2. Further, a pulse signal corresponding to the lens movementinstruction is output from the controller 320-2. The pulse signal outputfrom the controller 320-2 is input in the driver 320-4. Further, thedriver 320-4 drives the motor 320-6 according to the input pulse signal.Here, it is preferable that a stepping motor is used as the motor 320-6.Meanwhile, the pulse signal output from the controller 320-2 is alsosent to the up-down counter 320-8.

The up-down counter 320-8 counts the sent pulse signal and outputs thecount value. Here, since the up-down counter 320-8 is reset to 0 whenfocus lens FL reaches the HP by the HP sensor 322 as mentioned above,the count value of the up-down counter 320-8 shows the lens position offocus lens FL based on the HP.

<Point Restoration Processing>

FIG. 7 illustrates a main part block diagram of the main CPU 214 and thedigital signal processing unit 206 that perform point restorationprocessing. The main CPU 214 includes a lens position input unit 214-2,a first subject distance estimation unit 214-4, an estimation variationacquisition unit 214-6 and a second subject distance estimation unit214-8 as a function for restoration processing.

The lens position input unit 214-2 inputs the current count value of theup-down counter 320-8 after focusing control of focus lens FL, asinformation showing the lens position.

The first subject distance estimation unit 214-4 inputs the informationshowing the lens position from the lens position input unit 214-2, andestimates the distance of the subject (subject distance) focused byfocus lens FL of the current lens position. Here, since the lensposition of focus lens FL and the subject distance focused by the lensposition have a certain relationship, it is possible to estimate thesubject distance from the lens position of focus lens FL.

Information showing the subject distance (first subject distance)estimated by the first subject distance estimation unit 214-4 is outputto the estimation variation acquisition unit 214-6 and the secondsubject distance estimation unit 214-8.

The estimation variation acquisition unit 214-6 calculates the error ofthe first subject distance (estimation variation) on the basis of theinformation showing the first subject distance which is input from thefirst subject distance estimation unit 214-4.

Here, the estimation variation is variation of the first subjectdistance estimated from the lens position of focus lens FL, and it iscaused depending on the detection accuracy of the focusing stop positionof focus lens FL, and so on. For example, when the lens barrelcharacteristic changes by temperature, even if the subject distance isconstant, the focusing stop position of focus lens FL changes andincorrect measurement is performed (estimation variation occurs).

Moreover, as for the estimation variation, there is an error range(range of estimation variation) that extends to the infinity side andthe close end side around the first subject distance. The estimationvariation acquisition unit 214-6 calculates the maximum infinity-sideestimation variation in the range of estimation variation with respectto the first subject distance.

For example, in a case where the count value of the up-down counter320-8 from the HP (or INF) is different from the original count valueeven if the subject distance is constant (in a case where the focusingstop position of focus lens FL is different), the error of this countvalue is estimation variation. In a case of a lens apparatus in whichthe error of the count value occurs within the range of ±n pulses, themovement amount of focus lens FL for n pulses corresponds to the maximuminfinity-side estimation variation. Here, since the variation of thefocusing subject distance per pulse varies depending on the currentfocusing stop position of focus lens FL (first subject distance), theestimation variation acquisition unit 214-6 calculates the estimationvariation of the first subject distance on the basis of the informationshowing the first subject distance and information on the error of thecount value (n pulses).

Here, the error may be calculated on various conditions or selected froman error database stored in an error storage unit (not illustrated) thatis means for storing the error. For example, the error is unique toindividuals (individual differences) in an interchangeable lens ordepends on the temperature characteristic of the interchangeable lens.

For example, in the case of the imaging device 100 of theinterchangeable lens type, since the error range changes for eachinterchangeable lens, an estimation variation storage unit that is meansfor storing an error (estimation variation) may be included in theestimation variation acquisition unit 214-6 to store the error of eachinterchangeable lens (estimation variation). In this case, when the lensapparatus 300 is attached, information related to the lens apparatus 300is acquired, and a corresponding estimation variation is acquired fromthe estimation variation storage unit on the basis of the information.Here, the above-mentioned estimation variation storage unit may beinstalled in the lens apparatus 300.

Information showing the estimation variation calculated by theestimation variation acquisition unit 214-6 is output to the secondsubject distance estimation unit 214-8. The second subject distanceestimation unit 214-8 calculates the second subject distance adding theestimated first subject distance and the maximum error (estimationvariation) on the infinity side, and outputs information showing thecalculated second subject distance to a restoration filter selectionunit 206-4 of the digital signal processing unit 206.

In the digital signal processing unit 206, main parts related torestoration processing are a restoration processing unit 206-2, therestoration filter selection unit 206-4 and a restoration filter storageunit 206-6.

The restoration filter storage unit 206-6 stores multiple restorationfilters created on the basis of the point spread function of variouslenses including the lens apparatus 300, for each lens. Moreover,multiple restoration filters correspond to a combination of a subjectdistance, zoom magnification, f-number and image height (restorationregion), and are a lot of restoration filters. Here, for ease ofexplanation, the restoration filter storage unit 206-6 is assumed todiscretely store the restoration filters according to the subjectdistance.

The restoration filter selection unit 206-4 inputs the informationshowing the second subject distance from the second subject distanceestimation unit 214-8 and selects a restoration filter corresponding tothe second subject distance from the restoration filter storage unit206-6 on the basis of this input information. Here, in a case where therestoration filter corresponding to the second subject distance is notstored in the restoration filter storage unit 206-6, a restorationfilter on the infinity side closest to the second subject distance isselected. Here, details of a restoration filter selection method aredescribed later.

The restoration processing unit 206-2 performs restoration processingwith respect to image data by the use of the restoration filter selectedby the restoration filter selection unit 206-4.

Next, the restoration filter generation method is described.

First, a point image (point light source) is taken by the lens apparatus300 to measure a PSF (Point Spread Function) of the imaging lens of thelens apparatus 300 of the imaging device 100 at the time of adjustmentsuch as the time before the imaging device 100 is shipped, and a blur(the point image is expanded) image is acquired by aberration.

At this time, there are a method of using the imaging element 202dedicated for measurement and a method of using the imaging element 202actually incorporated in the imaging device 100. In the former case, itis suitable for the measurement of PSF corresponding to only the lensapparatus 300, and, in the latter case, it is suitable for themeasurement of PSF taking into account the influence of the imagingelement 202 (color filter, and so on) too.

Subsequently, when the blur image acquired by taking the point image byan imaging unit including the lens apparatus 300 and the imaging element202 is assumed to be G(X,Y), the original point image is assumed to beF(X,Y) and point spread function (PSF) is assumed to be H(X,Y), they canbe expressed by the following equation.G(X,Y)=H(X,Y)*F(X,Y)  [Equation 1]

Where * indicates convolution.

H(X,Y) of Equation 1 (that is, point spread function (PSF)) iscalculated on the basis of blur image G(X,Y) acquired by taking thepoint image as above.

Next, the inverse function of the above-mentioned calculated pointspread function (PSF) is calculated. When this inverse function isassumed to be R(X,Y), by performing convolution processing of imageG(X,Y) subjected to phase modulation like the following equation byR(X,Y), a restoration image corresponding to original image F(X,Y) isacquired (restoration processing).G(X,Y)*R(X,Y)=F(X,Y)  [Equation 2]

This R(X,Y) is called a restoration filter. As the restoration filter,it is possible to use the least square filter (Wiener filter) thatminimizes the square mean error between the original image and therestoration image, a limited reverse-convolution filter, a recursivefilter and a homomorphism filter, and so on.

First Embodiment

FIGS. 8 to 10 are diagrams to describe a way of storing and selecting arestoration filter in the first embodiment of the present invention,which is performed by the restoration filter selection unit 206-4 andthe restoration filter storage unit 206-6.

FIG. 8 is an image view of the point spread function (PSF) for each ofmultiple subject distances having a predetermined subject distanceinterval in a range from the close end (0.2 m) to the infinity (INF). Asmentioned above, a restoration filter is generated on the basis of thesePSFs.

As illustrated in FIG. 8, the PSF on the infinity side has smallerextent than the PSF on the close end side, and therefore restorationprocessing by a restoration filter on the infinity side is weakerrestoration processing than restoration processing by a restorationfilter on the close end side.

Moreover, in the example illustrated in FIG. 8, it is assumed thatrestoration filters corresponding to six distances (subject distances)of the close end (0.2 m), 0.25 m, 0.33 m, 0.5 m, 1 m and INF in a range(range in which imaging is possible) from the close end (0.2 m) to theinfinity are stored. Here, these subject distances are subject distancescorresponding to respective lens positions dividing the amount ofmovement from the close end of focus lens FL to INF into almost fiveequal parts.

The range shown by an arrow in FIG. 8 shows an estimation variation ineach subject distance in a case where 0.2 m, 0.25 m, 0.33 m, 0.5 m, 1 mor INF is estimated by the first subject distance estimation unit 214-4.

Moreover, a circle at an arrow point in the figure shows a distance(second subject distance) adding the first subject distance and themaximum infinity-side estimation variation in a range of estimationvariation with respect to the first subject distance in a case where thesubject distance (first subject distance) estimated by the first subjectdistance estimation unit 214-4 is 0.2 m, 0.25 m, 0.33 m, 0.5 m, 1 m orINF.

In FIG. 8, in a case where the first subject distance estimated by thefirst subject distance estimation unit 214-4 is 0.2 m, there is an errorrange from 0.25 m to 0.2 m, and the second subject distance estimated bythe second subject distance estimation unit 214-8 taking into accountthe maximum infinity-side estimation variation is 0.25 m. Further, arestoration filter on the infinity side closest to the second subjectdistance is used. That is, a restoration filter generated correspondingto a subject distance of 0.25 m is selected, and overcorrection isprevented. That is, it is assumed that the restoration filtercorresponding to the infinity side closest to the second subjectdistance is selected.

Here, the overcorrection in restoration processing is to estimate acloser subject distance than an actual subject distance and performrestoration processing by the use of a stronger (larger) restorationfilter than a restoration filter to be originally used. When theovercorrection in the restoration processing is performed, the qualityof an acquired image deteriorates.

As a concrete example in which overcorrection in restoration processingoccurs, in a case where the first subject distance estimated by thefirst subject distance estimation unit 214-4 is 0.25 m and the errorrange is from 0.33 m to 0.2 m, when a restoration filter generatedcorresponding to a subject distance of 0.25 m or 0.2 m is selected,there is a possibility of overcorrection in restoration processing. Thatis, since an actual subject distance is from 0.33 m to 0.2 m, in a casewhere the actual subject distance is 0.33 m, restoration processingusing a restoration filter generated corresponding to a subject distanceof 0.25 m or 0.2 m causes overcorrection.

Meanwhile, in a case where a restoration filter generated according to asubject distance that is greatly separated on the infinity side ascompared with the second subject distance, the effect of restorationprocessing cannot be acquired sufficiently. For example, whenrestoration processing is performed using a restoration filtercorresponding to INF though the second subject distance is 0.25 m, therestoration processing is weak, and it is not possible to sufficientlyacquire the effect of the restoration processing.

Therefore, in a case where the first subject distance estimated by thefirst subject distance estimation unit 214-4 is 0.25 m and the secondsubject distance estimated by the second subject distance estimationunit 214-8 is 0.33 m, it is possible to prevent the occurrence ofovercorrection by using a restoration filter generated corresponding toa subject distance of 0.33 m.

In FIG. 8, similar to a case where 0.25 m, 0.33 m, 0.5 m, 1 m or INF isestimated as the first subject distance, an arrow range shows the errorrange of the subject distance and, within an error range, a restorationfilter corresponding to the infinity side closest to the second subjectdistance is selected. Therefore, a restoration filter generatedcorresponding to a subject distance of 0.33 m is used in a case where0.25 m is estimated as the first subject distance, a restoration filtergenerated corresponding to a subject distance of 0.5 m is used in a casewhere 0.33 m is estimated, a restoration filter generated correspondingto a subject distance of 1 m is used in a case where 0.5 m is estimated,and a restoration filter generated corresponding to a subject distanceof INF is used in a case where INF is estimated.

Moreover, in a case where a subject distance different from 0.2 m, 0.25m, 0.33 m, 0.5 m, 1 m and INF is estimated as the first subjectdistance, for example, in a case where a subject distance between 0.33 mand 0.5 m is estimated, the second subject distance taking into accountthe maximum estimation variation is a subject distance between 0.5 m and1 m. Therefore, a restoration filter on the infinity side closest tothis second subject distance is a restoration filter generatedcorresponding to a subject distance of 1 m.

Moreover, in the imaging device 100, in photographing in a case where itis decided in advance that a distance place is photographed (scenerymode), a restoration filter generated corresponding to a subjectdistance of INF may be used regardless of the second subject distance.

FIG. 9 illustrates a measurement error or the like corresponding to alens (interchangeable lens) different from the lens apparatus 300, and,since the temperature characteristic varies depending on the lens typeand there are individual differences, estimation variation of thesubject distance estimated from the focusing stop position of the focuslens varies.

As compared with the case of FIG. 8, FIG. 9 illustrates a case whereestimation variation (error range) shown by an arrow range is large(long). In the case of FIG. 9, a restoration filter on the infinity sideclosest to the maximum possible subject distance from the second subjectdistance is used. That is, a restoration filter generated correspondingto a subject distance of 0.33 m is used in a case where 0.2 m isestimated as the first subject distance, a restoration filter generatedcorresponding to a subject distance of 0.5 m is used in a case where0.25 m is estimated, a restoration filter generated corresponding to asubject distance of 1 m is used in a case where 0.33 m is estimated, anda restoration filter generated corresponding to a subject distance ofINF is used in a case where 1 m or more is estimated.

As compared with the case of FIG. 9, FIG. 10 illustrates a case where anerror range is larger (longer). In the case of FIG. 10, even if theclose end (0.2 m) is estimated as the first subject distance, a subjectdistance (second subject distance) taking into account the estimationvariation is larger than 1 m. Therefore, a restoration filter that doesnot cause overcorrection is only a restoration filter generatedcorresponding to INF. That is, in the case of a lens having theestimation variation illustrated in FIG. 10, only a restoration filtergenerated corresponding to INF is used regardless of an estimatedsubject distance.

Second Embodiment

FIGS. 11 to 13 are diagrams to describe a way of storing and selecting arestoration filter in the second embodiment of the present invention.

Similar to the first embodiment illustrated in FIGS. 8 to 10, arestoration filter corresponding to a subject distance is stored in thesecond embodiment illustrated in FIGS. 11 to 13, but a way of storingthe restoration filter in the restoration filter storage unit 206-6 isdifferent in the second embodiment.

In the second embodiment, the restoration filter storage unit 206-6 thatis a restoration filter storage unit stores a corresponding restorationfilter within a range between the infinity and a subject distance(hereafter referred to as “estimation variation close range”) addingonly the maximum infinity-side estimation variation in a range ofestimation variation with respect to a nearest-side subject distance tothe nearest-side subject distance estimated by the first subjectdistance estimation unit 214-4.

There is a possibility that a restoration filter within a range on thenearer side than the estimation variation close range performsrestoration processing that causes overcorrection. Therefore, onlymultiple corresponding restoration filters within a range between theestimation variation close range and the infinity are stored in therestoration filter storage unit 206-6, such that a restoration filterwithin a range on the closer side than the estimation variation closerange is not stored from the beginning. By this means, the number ofrestoration filters to be stored in the restoration filter storage unit206-6 is reduced.

That is, while a restoration filter is stored in the restoration filterstorage unit 206-6 regardless of whether it is used for each of multiplesubject distances set beforehand in the first embodiment, a restorationfilter that is not used depending on a relationship with the error rangeof a subject distance is not stored in the restoration filter storageunit 206-6 in the second embodiment (see FIGS. 11 to 13).

FIG. 11 is a diagram illustrating the second embodiment corresponding tothe first embodiment illustrated in FIG. 8. When the second embodimentillustrated in FIG. 11 and the first embodiment illustrated in FIG. 8are compared, there is a difference in that a restoration filtercorresponding to a subject distance of 0.2 m is not provided in thesecond embodiment.

In the case of estimation variation shown by arrows in FIG. 11, even ifthe first subject distance is estimated to be the close end (0.2 m),taking into account an error range, the second subject distance(estimation variation close range) is 0.25 m and a restoration filtercorresponding to the infinity side closest to the estimation variationclose range is a restoration filter generated corresponding to a subjectdistance of 0.25 m. In other words, in a case where there is provided anerror range as illustrated in FIG. 11, a restoration filter generatedcorresponding to a subject distance of 0.2 m as illustrated in FIG. 8 isnot used regardless of how the first subject distance is. Further, therestoration filter that is not used is not stored in the restorationfilter storage unit 206-6.

FIG. 12 is a diagram illustrating the second embodiment corresponding tothe first embodiment illustrated in FIG. 9. When the second embodimentillustrated in FIG. 12 and the first embodiment illustrated in FIG. 9are compared, there is a difference in that restoration filterscorresponding to subject distances of 0.2 m and 0.25 m are not providedin the second embodiment.

In the case of estimation variation shown by arrows in FIG. 12, even ifthe first subject distance is estimated to be the close end (0.2 m),taking into account an error range, the second subject distance(estimation variation close range) is 0.33 m and a restoration filtercorresponding to the infinity side closest to the estimation variationclose range is a restoration filter generated corresponding to a subjectdistance of 0.33 m. In other words, in a case where there is provided anerror range as illustrated in FIG. 12, restoration filters generatedcorresponding to subject distances of 0.2 m and 0.25 m as illustrated inFIG. 9 are not used regardless of how the first subject distance is.Further, the restoration filters that are not used are not stored in therestoration filter storage unit 206-6.

FIG. 13 is a diagram illustrating the second embodiment corresponding tothe first embodiment illustrated in FIG. 10. When the second embodimentillustrated in FIG. 13 and the first embodiment illustrated in FIG. 10are compared, there is a difference in that restoration filterscorresponding to subject distances of 0.2 m, 0.25 m, 0.33 m, 0.5 m and 1m are not provided in the second embodiment.

In the case of estimation variation shown by arrows in FIG. 13, even ifthe first subject distance is estimated to be the close end (0.2 m),taking into account an error range, the second subject distance(estimation variation close range) is greater than 1 m and a restorationfilter corresponding to the infinity side closest to the estimationvariation close range is a restoration filter of INF(1). In other words,in a case where there is provided an error range as illustrated in FIG.13, restoration filters of (2) to (6) illustrated in FIG. 10 are notused regardless of how the first subject distance is. Further, therestoration filters of (2) to (6) that are not used are not stored inthe restoration filter storage unit 206-6, and the restoration filter of(1) selected depending on the relationship of the error range is storedin the restoration filter storage unit 206-6.

Thus, in the second embodiment illustrated in FIGS. 11 to 13, arestoration filter that is not used depending on the relationship withthe error range is not stored in the restoration filter storage unit206-6. By this means, it is possible to reduce the memory capacity tostore restoration filters.

Moreover, the restoration filter storage unit that is a restorationfilter storage unit has multiple tables in which multiple restorationfilters are stored corresponding to estimation variation, and therestoration filter selection unit may select a corresponding table frommultiple tables according to the magnitude of estimation variationacquired by the estimation variation acquisition unit 214-6 and select arestoration filter from the selected table.

FIG. 14 illustrates the appearance of a smartphone 500 that is anotherembodiment of the imaging device 100. The smartphone 500 illustrated inFIG. 14 has a flat chassis 502 and includes a display input unit 520 inwhich a display panel 521 as the monitor 212 and an operation panel 522as an input unit are integrated on one surface of the chassis 502.Moreover, the chassis 502 includes a speaker 531, a microphone 532, anoperation unit 540 and a camera unit 541. Here, the configuration of thechassis 502 is not limited to this, and, for example, it is possible toadopt a configuration in which the monitor 212 and the input unit areindependent, or a configuration having a folded structure or a slidingmechanism.

FIG. 15 is a block diagram illustrating the configuration of thesmartphone 500 illustrated in FIG. 14. As illustrated in FIG. 15, aradio communication unit 510, the display input unit 520, a call unit530, the operation unit 540, the camera unit 541, a storage unit 550, anexternal input/output unit 560, a GPS (Global Positioning System)reception unit 570, a motion sensor unit 580, a power supply unit 590and a main control unit 501 are included as main components of thesmartphone. Moreover, a radio communication function to perform mobileradio communication through base station apparatus BS and mobilecommunications network NW is included as a main function of thesmartphone 500.

The radio communication unit 510 performs radio communication with basestation apparatus BS accommodated in mobile communications network NWaccording to an instruction of the main control unit 501. By the use ofthis radio communication, various kinds of file data such as voice dataand image data, and email data, and so on, are transmitted and received,and web data and streaming data, and so on, are received.

The display input unit 520 is a so-called touch panel that displaysimages (still image and moving image) and character information, and soon, visually transmits information to the user and detects useroperation with respect to the displayed information by control of themain control unit 501, and includes the display panel 521 and theoperation panel 522. In a case where a generated 3D image isappreciated, it is preferable that the display panel 521 is 3D displaypanel.

As for the display panel 521, an LCD (Liquid Crystal Display) and anOELD (Organic Electro-Luminescence Display), and so on, are used as adisplay device. The operation panel 522 is a device that is placed so asto be able to visually confirm an image displayed on the display surfaceof the display panel 521 and that detects one or multiple coordinatesoperated by user's finger or stylus. When this device is operated byuser's finger or stylus, a detection signal generated depending on theoperation is output to the main control unit 501. Next, the main controlunit 501 detects the operation position (coordinates) on the displaypanel 521 on the basis of the received detection signal.

As illustrated in FIG. 14, the display panel 521 and the operation panel522 of the smartphone 500 integrally form the display input unit 520,and they are disposed such that the operation panel 522 completelycovers the display panel 521. In a case where this disposition isadopted, the operation panel 522 may have a function to detect useroperation even in a region outside the display panel 521. In otherwords, the operation panel 522 may include a detection region for anoverlapping part that overlaps with the display panel 521 (hereafterreferred to as “display region”), and a detection region for theremaining outer peripheral part that does not overlap with the displaypanel 521 (hereafter referred to as “non-display region”).

Here, the size of the display region and the size of the display panel521 may be completely matched, but it does not have to necessarily matchboth of them. Moreover, the operation panel 522 may include two sensingregions of the outer peripheral part and the remaining inner part. Inaddition, the width of the outer peripheral part is arbitrarily designedaccording to the size of the chassis 502, and so on. Furthermore, as aposition detection system adopted in the operation panel 522, there area matrix switch system, a resistance film system, a surface elastic wavesystem, an infrared ray system, an electromagnetic induction system andan electrostatic capacity system, and so on, and any system can beadopted.

The call unit 530 includes the speaker 531 and the microphone 532,converts user's voice input through the microphone 532 into voice datathat can be processed in the main control unit 501, outputs it to themain control unit 501, decodes voice data received by the radiocommunication unit 510 or the external input/output unit 560 and outputsit from the speaker 531. Moreover, as illustrated in FIG. 14, forexample, it is possible to mount the speaker 531 to the same surface asa surface on which the display input unit 520 is installed, and mountthe microphone 532 to the side surface of the chassis 502.

The operation unit 540 is a hardware key using a key switch, and so on,and accepts an instruction from the user. For example, as illustrated inFIG. 14, the operation unit 540 is mounted to the lower part or lowerside surface of the display unit of the chassis 502 of the smartphone500, and is a push-button switch that is turned on when being pressed bya finger, and so on, and enters an OFF state by the restoring force of aspring or the like when the finger is separated.

The storage unit 550 stores a control program and control data of themain control unit 501, address data associating the name and telephonenumber of the communication party, data of transmitted and receivedemail, web data downloaded by web browsing and downloaded content data,or temporarily stores streaming data, and so on. Moreover, the storageunit 550 includes an internal storage unit 551 incorporated in thesmartphone and an external storage unit 552 having a detachable externalmemory slot. Here, each of the internal storage unit 551 and theexternal storage unit 552 forming the storage unit 550 is realized usingstorage media such as a memory (for example, a Micro SD (registeredtrademark) memory, and so on) of a flash memory type, hard disk type,multimedia card micro type or card type, a RAM (Random Access Memory)and a ROM (Read Only Memory).

The external input/output unit 560 plays the role of an interface withall external devices coupled with the smartphone 500, and is directly orindirectly connected with other external devices by communication or thelike (for example, a universal serial bus (USB) and IEEE1394, and so on)or a network (for example, the Internet, wireless LAN, Bluetooth(registered trademark), RFID (Radio Frequency Identification), infraredcommunication (IrDA: Infrared Data Association) (registered trademark),UWB (Ultra Wideband) (registered trademark) and ZigBee (registeredtrademark), and so on).

As an external device coupled with the smartphone 500, for example,there are a wired/wireless headset, a wired/wireless external charger, awired/wireless data port, a memory card or SIM (Subscriber IdentityModule Card)/UIM (User Identity Module Card) card connected through acard socket, an external audio/video device connected through anaudio/video I/O (Input/Output) terminal, an external audio/video devicesubjected to wireless connection, a smartphone subjected towired/wireless connection, a personal computer subjected towired/wireless connection, a PDA subjected to wired/wireless connection,a personal computer subjected to wired/wireless connection and anearphone, and so on. The external input/output unit can transfer datatransmitted from such an external device to each component in thesmartphone 500 and transmit data in the smartphone 500 to the externaldevice.

The GPS reception unit 570 receives GPS signals transmitted from GPSsatellites ST1 to STn according to an instruction of the main controlunit 501, performs positioning calculation processing based on thereceived multiple GPS signals, and detects a position formed with thelatitude, longitude and altitude of the smartphone 500. The GPSreception unit 570 can detect the position by the use of positioninformation when the position information can be acquired from the radiocommunication unit 510 or the external input/output unit 560 (forexample, wireless LAN).

For example, the motion sensor unit 580 includes a three-axisacceleration sensor, and so on, and detects the physical movement of thesmartphone 500 according to an instruction of the main control unit 501.By detecting the physical movement of the smartphone 500, the movementdirection and acceleration of the smartphone 500 are detected. Thisdetection result is output to the main control unit 501.

The power supply unit 590 supplies power accumulated in a battery (notillustrated) to each part of the smartphone 500 according to aninstruction of the main control unit 501.

The main control unit 501 includes a microprocessor, performs operationaccording to a control program and control data stored in the storageunit 550, and integrates and controls each part of the smartphone 500.Moreover, to perform voice communication and data communication throughthe radio communication unit 510, the main control unit 501 has a mobilecommunication control function to control each part of a communicationsystem and an application processing function.

The application processing function is realized by making the maincontrol unit 501 perform operation according to application softwarestored in the storage unit 550. As the application processing function,for example, there are an infrared communication function to control theexternal input/output unit 560 and perform data communication with anopposite device, an email function to transmit and receive email, a webbrowsing function to browse web pages, and a function to generate a 3Dimage from a 2D image according to the present invention, and so on.

Moreover, the main control unit 501 has an image processing function todisplay an image on the display input unit 520, and so on, on the basisof image data (data of a still image and moving image) such as receptiondata and downloaded streaming data. The image processing functiondenotes a function that the main control unit 501 decodes theabove-mentioned image data, performs image processing on this decodingresult and displays an image on the display input unit 520.

In addition, the main control unit 501 performs display control for thedisplay panel 521 and operation detection control to detect useroperation through the operation unit 540 and the operation panel 522.

By performing display control, the main control unit 501 displays anicon to activate application software and a software key such as ascroll bar, or displays a window to create email. Here, the scroll bardenotes a software key to accept an instruction to move an image displaypart of a large image that cannot be settled in the display region ofthe display panel 521.

Moreover, by performing operation detection control, the main controlunit 501 detects user operation through the operation unit 540, acceptsoperation with respect to the above-mentioned icon or an input of acharacter string with respect to an input column of the above-mentionedwindow through the operation panel 522 or accepts a scroll request of adisplay image through a scroll bar.

In addition, the main control unit 501 has a touch panel controlfunction to: determine whether an operation position with respect to theoperation panel 522 is an overlapping part (display region) thatoverlaps with the display panel 521 or it is the remaining outerperipheral part (non-display region) that does not overlap with thedisplay panel 521 by performing operation detection control; and controlthe sensing region of the operation panel 522 and the display positionof the software key.

Moreover, the main control unit 501 can detect gesture operation withrespect to the operation panel 522 and execute a preset functionaccording to the detected gesture operation. The gesture operation isnot simple touch operation in the related art, and means operation todraw a trajectory by finger and so on, designate multiple positions atthe same time or combine these to draw a trajectory of at least one ofmultiple positions.

The camera unit 541 is a digital camera that performs electronicphotographing by the use of the imaging element 202 such as a CMOS(Complementary Metal Oxide Semiconductor) and a CCD (Charge-CoupledDevice), and has a function similar to the imaging device 100illustrated in FIG. 1, and so on.

That is, the camera unit 541 is configured so as to be able to switch amanual focus mode and an auto focus mode, and, when the manual focusmode is selected, it is possible to perform focusing of the imaging lensof the camera unit 541 by operating an icon button for focus or the likedisplayed on the operation unit 540 or the display input unit 520.Further, at the time of the manual focus mode, a live view imagecombining split images is displayed on the display panel 521, andthereby it is possible to confirm the focusing state at the time ofmanual focus.

Moreover, by control of the main control unit 501, for example, thecamera unit 541 can convert image data acquired by imaging intocompressed image data of JPEG (Joint Photographic coding Experts Group)or the like, record it in the storage unit 550 and output it through theexternal input/output unit 560 or the radio communication unit 510. Inthe smartphone 500 illustrated in FIG. 14, the camera unit 541 ismounted to the same surface as the display input unit 520, but themounting position of the camera unit 541 is not limited to this, and itmay be mounted to the back surface of the display input unit 520 ormultiple cameras 541 may be mounted. Here, in a case where multiplecameras 541 are mounted, it is possible to switch the camera unit 541provided for photographing and perform photographing alone, or performphotographing by the use of multiple cameras 541 at the same time.

Moreover, the camera unit 541 can be used for various functions of thesmartphone 500. For example, it is possible to display an image acquiredin the camera unit 541 on the display panel 521 and use an image of thecamera unit 541 as one of operation inputs of the operation panel 522.Moreover, when the GPS reception unit 570 detects a position, it ispossible to detect the position with reference to an image from thecamera unit 541. Furthermore, with reference to the image from thecamera unit 541, it is possible to determine the optical axis directionof the camera unit 541 of the smartphone 500 and determine the currentuse environment, without using a three-axis acceleration sensor or whileusing the three-axis acceleration sensor together. It is natural that itis possible to use the image from the camera unit 541 in applicationsoftware.

Besides, position information acquired by the GPS reception unit 570,voice information (which may be text information by performing voicetext conversion by the main control unit, and so on) acquired by themicrophone 532 and posture information acquired by the motion sensorunit 580, and so on, can be added to image data of a still image ormoving image and recorded in the storage unit 550 or output through theexternal input/output unit 560 or the radio communication unit 510.

Moreover, it is needless to say that the present invention is notlimited to the above-mentioned embodiments and various changes arepossible without departing from the scope of the present invention.

What is claimed is:
 1. An imaging device comprising: an imaging unitconfigured to capture a subject image formed by an imaging lens andgenerate an image including the subject image; a first subject distanceestimation unit configured to calculate an estimated subject distance ofthe subject focused on by a focus detection unit and in the subjectimage captured by the imaging unit; a restoration filter storage unitconfigured to store multiple restoration filters created based on atleast a point spread function corresponding to a subject distance; anestimation variation acquisition unit configured to acquire a maximuminfinity-side estimation variation in a range of estimation variationwith respect to the subject distance estimated by the first subjectdistance estimation unit; a second subject distance estimation unitconfigured to calculate a subject distance adding the maximuminfinity-side estimation variation acquired by the estimation variationacquisition unit to the subject distance estimated by the first subjectdistance estimation unit; a restoration filter selection unit configuredto select a restoration filter corresponding to an infinity side closestto the subject distance calculated by the second subject distanceestimation unit among the multiple restoration filters stored in therestoration filter storage unit; and a restoration processing unitconfigured to perform restoration processing of the image acquired bythe imaging unit using the restoration filter selected by therestoration filter selection unit.
 2. The imaging device according toclaim 1, wherein the first subject distance estimation unit estimatesthe subject distance based on a lens position of a focus lens of theimaging lens.
 3. The imaging device according to claim 2, wherein: theimaging lens is an interchangeable lens; and the estimation variationacquisition unit has an estimation variation storage unit configured tostore the maximum infinity-side estimation variation of theinterchangeable lens for each interchangeable lens, acquires informationon the interchangeable lens to be attached, and acquires a correspondingestimation variation from the estimation variation storage unit based onthe acquired information on the interchangeable lens.
 4. The imagingdevice according to claim 2, wherein: the imaging lens is a lens havingan estimation variation storage unit configured to store the maximuminfinity-side estimation variation in the range of estimation variationwith respect to the subject distance estimated by the first subjectdistance estimation unit; and the estimation variation acquisition unitacquires the estimation variation from the estimation variation storageunit of the lens.
 5. The imaging device according to claim 2, whereinthe maximum infinity-side estimation variation in the range ofestimation variation with respect to the subject distance estimated bythe first subject distance estimation unit is at least one of estimationvariation by an individual difference of an interchangeable lens andestimation variation by a temperature characteristic of theinterchangeable lens.
 6. The imaging device according to claim 1,wherein the restoration filter storage unit stores multiplecorresponding restoration filters in a range between an infinity and asubject distance adding only the maximum infinity-side estimationvariation in the range of estimation variation with respect to anearest-side subject distance to the nearest-side subject distanceestimated by the first subject distance estimation unit.
 7. The imagingdevice according to claim 1, wherein the restoration filter storage unitstores the multiple restoration filters corresponding to a subjectdistance regardless of the imaging lens.
 8. The imaging device accordingto claim 1, wherein the imaging lens is an interchangeable lens in whichthe restoration filter storage unit is incorporated.
 9. The imagingdevice according to claim 1, further comprising a recording unitconfigured to record an image acquired by performing restorationprocessing by the restoration processing unit.
 10. The imaging deviceaccording to claim 1, wherein: the restoration filter storage unit hasmultiple tables in which multiple restoration filters are storedcorresponding to the estimation variation; and the restoration filterselection unit selects a corresponding table from the multiple tablesaccording to magnitude of the estimation variation acquired by theestimation variation acquisition unit, and selects the restorationfilter from the selected table.
 11. An image processing methodcomprising: an image generation step of generating an image including asubject image from an imaging unit; a first subject distance estimationstep of calculating an estimated subject distance of the subject focusedon by a focus detection unit and in the subject image captured by theimaging unit; a step of preparing a restoration filter storage unitconfigured to store multiple restoration filters created based on atleast a point spread function corresponding to a subject distance; anestimation variation acquisition step of acquiring a maximuminfinity-side estimation variation in a range of estimation variationwith respect to the subject distance estimated in the first subjectdistance estimation step; a second subject distance estimation step ofcalculating a subject distance adding the maximum infinity-sideestimation variation acquired in the estimation variation acquisitionstep to the subject distance estimated in the first subject distanceestimation step; a restoration filter selection step of selecting arestoration filter corresponding to an infinity side closest to thesubject distance calculated in the second subject distance estimationstep among the multiple restoration filters stored in the restorationfilter storage unit; and a restoration processing step of performingrestoration processing of the image acquired by the imaging unit usingthe restoration filter selected in the restoration filter selectionstep.